Skip to content

Node v18.15.0 and higher ignore --max-http-header-size #47246

New issue
Jump to bottom
New issue
Jump to bottom
Closed
nodejs/undici
#2234
Closed
Node v18.15.0 and higher ignore --max-http-header-size#47246
nodejs/undici
#2234
Labels
httpIssues or PRs related to the http subsystem.
@havran

Description

@havran
havran
opened on Mar 24, 2023

Version

v18.15.0 and v19.8.1

Platform

Linux FESK-LTP0007 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux (Ubuntu 22.04)

Subsystem

No response

What steps will reproduce the bug?

I use urql in my NextJS application, to communicate with backend CMS. For some query CMS send extra large headers with response (around 22kb).

I use node 18 as runner for project where I use URQL library to GraphQL request to backend CMS, but there I get console error:

[newro-15509] Network Error! CombinedError: [Network] fetch failed
    at makeErrorResult (file:///data/***/node_modules/@urql/core/dist/urql-core-chunk.mjs:284:10)
    at file:///data/***/node_modules/@urql/core/dist/urql-core-chunk.mjs:462:15 {
  graphQLErrors: [],
  networkError: TypeError: fetch failed
      at Object.fetch (node:internal/deps/undici/undici:11457:11) {
    cause: HeadersOverflowError: Headers Overflow Error
        at Parser.trackHeader (node:internal/deps/undici/undici:9453:37)
        at Parser.onHeaderValue (node:internal/deps/undici/undici:9448:14)
        at wasm_on_header_value (node:internal/deps/undici/undici:9257:34)
        at wasm://wasm/00036ac6:wasm-function[49]:0xf91
        at wasm://wasm/00036ac6:wasm-function[68]:0x7316
        at wasm://wasm/00036ac6:wasm-function[67]:0x154d
        at wasm://wasm/00036ac6:wasm-function[21]:0x552
        at Parser.execute (node:internal/deps/undici/undici:9373:26)
        at Parser.readMore (node:internal/deps/undici/undici:9352:16)
        at Socket.onSocketReadable (node:internal/deps/undici/undici:9660:14) {
      code: 'UND_ERR_HEADERS_OVERFLOW'
    }
  },
  response: undefined
}

I investigate and I found solution to set http header size by using environment: NODE_OPTIONS=--max-http-header-size=32768

This seems to be ignored for Node version 18 and higher (I also try node 19). On Node 16, this environment variable solve error.

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

No response

What do you see instead?

I see error message:

[newro-15509] Network Error! CombinedError: [Network] fetch failed
    at makeErrorResult (file:///data/***/node_modules/@urql/core/dist/urql-core-chunk.mjs:284:10)
    at file:///data/***/node_modules/@urql/core/dist/urql-core-chunk.mjs:462:15 {
  graphQLErrors: [],
  networkError: TypeError: fetch failed
      at Object.fetch (node:internal/deps/undici/undici:11457:11) {
    cause: HeadersOverflowError: Headers Overflow Error
        at Parser.trackHeader (node:internal/deps/undici/undici:9453:37)
        at Parser.onHeaderValue (node:internal/deps/undici/undici:9448:14)
        at wasm_on_header_value (node:internal/deps/undici/undici:9257:34)
        at wasm://wasm/00036ac6:wasm-function[49]:0xf91
        at wasm://wasm/00036ac6:wasm-function[68]:0x7316
        at wasm://wasm/00036ac6:wasm-function[67]:0x154d
        at wasm://wasm/00036ac6:wasm-function[21]:0x552
        at Parser.execute (node:internal/deps/undici/undici:9373:26)
        at Parser.readMore (node:internal/deps/undici/undici:9352:16)
        at Socket.onSocketReadable (node:internal/deps/undici/undici:9660:14) {
      code: 'UND_ERR_HEADERS_OVERFLOW'
    }
  },
  response: undefined
}

Additional information

No response

Activity

deokjinkim
changed the title Node v18.15.0 and gigher ignore --max-http-header-size Node v18.15.0 and higher ignore --max-http-header-size on Mar 24, 2023
deokjinkim
added
httpIssues or PRs related to the http subsystem.
on Mar 24, 2023
bnoordhuis

bnoordhuis commented on Mar 25, 2023

@bnoordhuis
bnoordhuis
on Mar 25, 2023
Member

Can you include a test case? No third-party modules, just built-in modules.

I'm reasonably sure the option gets passed to undici (the new http library) because we have tests that exercise the flag.

havran

havran commented on Mar 25, 2023

@havran
havran
on Mar 25, 2023 · edited by havran
Author

Ok, I just create wery simple HTTP server:

# server.js

const http = require("http");

const host = 'localhost';
const port = 8000;

const requestListener = function (req, res) {
    res.setHeader("Content-Type", "application/json");
    res.setHeader("X-FakeSize", new Array(20000).join('a'));
    res.writeHead(200);
    res.end("My first server!");
};

const server = http.createServer(requestListener);
server.listen(port, host, () => {
    console.log(`Server is running on http://${host}:${port}`);
});

I run server node server.js and in another window I run NODE_OPTIONS=--max-http-header-size=32768 node and on prompt I use simple fetch request:

$ NODE_OPTIONS=--max-http-header-size=32768 node
Welcome to Node.js v18.15.0.
Type ".help" for more information.
> fetch('http://localhost:8000').then(res => console.log(res));
Promise {
  <pending>,
  [Symbol(async_id_symbol)]: 60,
  [Symbol(trigger_async_id_symbol)]: 27
}
> Uncaught TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11413:11) {
  cause: HeadersOverflowError: Headers Overflow Error
      at Parser.trackHeader (node:internal/deps/undici/undici:9411:37)
      at Parser.onHeaderValue (node:internal/deps/undici/undici:9406:14)
      at wasm_on_header_value (node:internal/deps/undici/undici:9215:34)
      at wasm://wasm/00036ac6:wasm-function[49]:0xf91
      at wasm://wasm/00036ac6:wasm-function[68]:0x7316
      at wasm://wasm/00036ac6:wasm-function[67]:0x154d
      at wasm://wasm/00036ac6:wasm-function[21]:0x552
      at Parser.execute (node:internal/deps/undici/undici:9331:26)
      at Parser.readMore (node:internal/deps/undici/undici:9310:16)
      at Socket.onSocketReadable (node:internal/deps/undici/undici:9618:14) {
    code: 'UND_ERR_HEADERS_OVERFLOW'
  }
}
>

Node version used:

$ node --version
v18.15.0

os

 neofetch
            .-/+oossssoo+/-.               havran@SGC
        `:+ssssssssssssssssss+:`           ----------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 20.04.5 LTS on Windows 10 x86_64
    .ossssssssssssssssssdMMMNysssso.       Kernel: 5.15.90.1-microsoft-standard-WSL2
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Uptime: 23 mins
  +ssssssssshmydMMMMMMMNddddyssssssss+     Packages: 1234 (dpkg), 19 (brew)
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Shell: bash 5.0.17
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Theme: Adwaita [GTK3]
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Icons: Adwaita [GTK3]
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Terminal: Relay(2291)
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: AMD Ryzen 7 1700 (8) @ 2.994GHz
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   GPU: 43ca:00:00.0 Microsoft Corporation Device 008e
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Memory: 883MiB / 7945MiB
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.
havran

havran commented on Mar 25, 2023

@havran
havran
on Mar 25, 2023
Author

I also attach image where I test node --max-http-header-size=32768, seems it is works, http.maxHeaderSize is set correct, but fetch request still show UND_ERR_HEADERS_OVERFLOW:

image

climba03003

climba03003 commented on Mar 25, 2023

@climba03003
climba03003
on Mar 25, 2023 · edited by climba03003
Contributor

undici does not use http.maxHeaderSize as the default.
It has it's own default value and you need to change it yourself.

https://github.com/nodejs/undici/blob/e6fc80f809d1217814c044f52ed40ef13f21e43c/lib/client.js#L225

You can change the behavior by

const { Agent, setGlobalDispatcher } = require('undici')

setGlobalDispatcher(new Agent({
  maxHeaderSize: http.maxHeaderSize
}))

Full code that would work for your usage.

const http = require("http");
const { fetch, Agent, setGlobalDispatcher } = require('undici')

setGlobalDispatcher(new Agent({
  maxHeaderSize: http.maxHeaderSize
}))

const host = 'localhost';
const port = 8000;

const requestListener = function (req, res) {
    res.setHeader("Content-Type", "application/json");
    res.setHeader("X-FakeSize", new Array(20000).join('a'));
    res.writeHead(200);
    res.end("My first server!");
};

const server = http.createServer(requestListener);
server.listen(port, host, () => {
    console.log(`Server is running on http://${host}:${port}`);

    fetch(`http://${host}:${port}`)
      .then(function() {
        console.log('normal')
        process.exit(0)
      })
      .catch(function(err) {
        console.log('error')
        process.exit(1)
      })
});
bnoordhuis

bnoordhuis commented on Mar 26, 2023

@bnoordhuis
bnoordhuis
on Mar 26, 2023
Member

undici does not use http.maxHeaderSize as the default.

@nodejs/http if ☝️is true, isn't that a bug?

mcollina

mcollina commented on Mar 26, 2023

@mcollina
mcollina
on Mar 26, 2023
SponsorMember

Given it's a read-only value, I think undici can use that value as the default when creating the default Agent.

Shall we move this issue to undici?

havran

havran commented on Mar 26, 2023

@havran
havran
on Mar 26, 2023
Author

IMHO this behaviour should be same than in older node versions. Also when user use frameworks like NextJS, then is unexpected when there is no simple way change this value.

adoprog

adoprog commented on Aug 23, 2023

@adoprog
adoprog
on Aug 23, 2023

Just got this error in NextJS after upgrading to its recent version. Is there any way to apply the workaround to it?

adoprog
mentioned this on Aug 31, 2023
balazsorban44

balazsorban44 commented on Aug 31, 2023

@balazsorban44
balazsorban44
on Aug 31, 2023 · edited by balazsorban44

My two cents here, @mcollina's #47246 (comment) makes a lot of sense to me. In newer versions of Node.js, I consider undici as an implementation detail of globalThis.fetch, and any Node.js flag should configure fetch accordingly.

I don't mind if it's a different flag, even something like --undici-max-http-header-size, but it should be configurable without importing undici.

Ethan-Arrowood

Ethan-Arrowood commented on Aug 31, 2023

@Ethan-Arrowood
Ethan-Arrowood
on Aug 31, 2023
Contributor

For this particular problem, users can use setGlobalDispatcher to modify this value and it should work. But I agree that since that method is not documented well; undici should start respecting more of Node's http flags/options.

It seems @mcollina is also in agreement; so lets ship it 🚢

climba03003

climba03003 commented on Aug 31, 2023

@climba03003
climba03003
on Aug 31, 2023
Contributor

I believe it just the matter of will someone willing to open PR to update undici default value to http.maxHeaderSize.

balazsorban44
mentioned this on Aug 31, 2023
balazsorban44

balazsorban44 commented on Oct 14, 2023

@balazsorban44
balazsorban44
on Oct 14, 2023

Hi everyone, this has shipped in Node.js, you can upgrade, and will be able to set this flag, I believe the issue can be closed @havran.

bnoordhuis
closed this as completedon Oct 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    httpIssues or PRs related to the http subsystem.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    Participants

    @mcollina@bnoordhuis@havran@adoprog@deokjinkim

    Issue actions
      Node v18.15.0 and higher ignore --max-http-header-size · Issue #47246 · nodejs/node